ClariCGI Template Language

In order to allow local sites and group administrators to customize the look and feel of the web pages output by ClariCGI, it features a programming language that is used to define "templates" for most of the web output it does.

A template is a web page with special tags inside that say things like, "insert the headline of the article here."

This lets admins write a page of HTML to lay out the look of an article or menu of articles with the ability to put in placeholders where the headline, byline, article text and various other components might go.

In fact, however, this system is a great deal more powerful than that. While most users may simply use it in this "fill in the placeholder" method, it actually includes a fairly full featured programming language that allows sophisticated users to exert a great deal of control over the contents of web pages and how they are layed out. Most of the various components of a page are put into "variables" so they can be manipulated. Control allows administrators to generate things like:

1. Different pages for different browsers
2. Conditional text that appears or doesn't based on other text or flags
3. Rearrangement or editing of text from variables
4. Database fetches based on variables
5. The insertion of different or modified article queries into pages
6. Fine control of newsgroup menus or the individual look of any query.

Generally the use of this template language is restricted to local administrators and those they trust. While in theory the language can't do any significant system operations, it is always a high risk to allow arbitrary users to cause a web CGI to execute programs in a programming language of any sophistication. This means all templates must be pre-prepared and stored in the ClariCGI or USENET directories.

ClariNet provides some sample templates for newsgroups and articles and you can take these or modify them as you see fit. For non-Clarinet groups, we expect to see other USENET participants eventually build templates for their favourite groups to give them a web look. Those templates will be distributed through trusted sources.

Template Types

Templates are used primarily to control the look of two things: Menus of stories as returned from the results of queries within a newsgroup, and the actual articles when they are read.

A template for a menu of stories is known as a newsgroup template, in particular because the most common purpose for one is to lay out the display of the latest articles in a newsgroup from the standard "show me this newsgroup" query in ClariCGI.

Templates can be found in several places. ClariCGI keeps a "DBM" style database, and it is most efficient, in terms of disk operations. However, ClariCGI will also pull templates from a special template directory and from newsgroup and newsgroup hierarchy directories as well.

For example the file group.template in the spool (or overview) directory for "clari.tw" will define the look of menus of articles in that newsgroup and any subgroup or hierarchy that doesn't have its own template. When presenting a group, ClariCGI tries to open a template file for the specific group, but if it doesn't find one it goes up the hierarchies to find one, or uses a default if none are found. A template at the root of the USENET news spool (or clari hierarchy spool) will define a default for that entire tree. (You can also do that in the config file.)

In this case, this allows easy maintenance of different templates for different newsgroups and hierarchies of newsgroups. For best efficiency, a tool is available with ClariCGI (claridb/build.perl) to build a DBM database from template files kept in newsgroup spools.

(ClariCGI actually uses the same directory that the "overview" file is kept in. That's usually the news spool, but some systems keep a parallel tree for such directories, and this is used by ClariCGI as well.)

The typical process for defining your own custom look for ClariNet or other articles is to define a basic template to be stored in the "clari" directory, and then any specific templates you would like for sub-hierarchies and specific groups. Some sites may want just one master template, some may want a custom look for every newsgroup.

Template Basics

A template consists of a stream of text with embedded "code snippets" which you may think of as placeholders. These are enclosed in braces.

If you want a left brace in your HTML, use the HTML escape sequence "{" in your template file. ClariCGI doesn't interpret this, all web browsers do, however. This

In this example, the contents of the variable "some_data" will be inserted into the stream of text in the paragraph.

In a template file everything between a pair of braces is a program in the ClariCGI language. The simplest program is a single "expression" and the simplest useful expression is a variable name, as shown in the example above. You can do a lot with just this, the insertion of variables into your HTML.

A program can also consist of one or more "statements." Statements are usually terminated by a semi-colon, so that's how you put several in a program. You can also just have a series of programs in a row, the effect is about the same and depends on your taste.

As noted, the simplest statement is an "expression" all by itself. This causes the value of the expression to be calculated and inserted into the text stream. We call this "printing." (Later it will be important to distinguish between certain parts of the language that "print" directly to the text stream and ones which return a value that can be printed when using this simple statement form.

While most statements (except an IF-block) need a semicolon after them, as a special exception a program that consists of a single expression doesn't need such a semicolon. Thus {some_data} is valid, as is {some_data;}.

From now on, when showing expressions, we usually won't show them bounded by braces, unless the example shows whole programs or shows expressions in the context of HTML templates

There are a variety of other statements, but inserting an expression into your text stream is by far the most common thing to do. The one gotcha is that the text is inserted as is, so variables containing "<" and ">" and "&" had better be HTML. There is a way to make sure those are expanded, see below.

How to use them?

The simplest way to use these templates is to take one of our sample templates from the template variables page and try it out with a specific newsgroup. For example, go to that newsgroup's spool directory (or overview directory if it has one) and copy our "sample group template" to the file group.template in that directory.

Now modify the file to your taste. Perhaps insert some extra HTML, some icons or a new layout using tables. Perhaps try our table based template or alter the look in any way you see fit. Once you are happy with the look, consider making the template apply to an entire hierarchy or modify it for other groups.

Variables

Variables can contain strings or numbers. These can be interchanged. If a number is used where a string is needed, it is converted to a digit string, except zero is converted to a null string. If a string is needed where a number is expected, the string is tested to see if it looks like a number, otherwise zero is used. In a "boolean" context (such as IF) a non-zero number is true, zero is false. A non-null string is true and a null string is false, with the important exception that the string "0" is considered false.

Variables can be permanent or temporary. Variables created by the template programs are usually temporary. System variables are usually permanent. Temporary variables disappear after the processing of a single article or newsgroup. This is mostly relevant for personal newspaper processing or multi-queries, where several newsgroups or articles can appear on one page.

Variables can be owned by the system, in which case template programs can't change their value. This is true of most configuration values and many other system values.

Variable names are case sensative, must start with a letter or _, and can consist of alphanumerics or the underbar character.

Unassigned variables are zero (null-string). No error is generated.

A guide to predefined variables is available. This guide also has descriptions of the existing templates and examples of how to use them.

Variables defined in a query must start with an upper-case "U." For that reason it is advised that template programers not use such variables or assume they start out null, since any user can define a variable starting with "U" in their query. On the other hand, this lets you allow queries to provide values to template programs.

Statements

Most ClariCGI programs are just expressions, and their value is output. More sophisticated users however will make use of some of the special statements in the language. If that's not you, you an skip this part.

The other statements you can perform are:

Interpolation Expression

% String-Expression

In this case, the expression (always a string) will be checked for variable names surrounded in braces. The string will be printed with the variable names inserted into it. Note that the string is printed, not returned as a new string. Note as well that only variable names can be interpolated, not full expressions, though this may change in future versions.

Note as well that any HTML special characters in the variable, such as "<", ">" and "&" will be expanded to HTML escape sequences. If you don't want that, you will need to build strings another way.

This statement simply allows a touch of the ability to output a string of HTML with several variables in it inside a statement, usually an IF statement. It makes for more readable templates. If you are not inside an IF statement, you can just interpolate variables with the general template language at its top levels.

Note that in interpolation, numeric zero variables are interpolated as "0", not as the null string.

Discarded expression

: Expression

This causes the expression to be calculated, but its results are discarded. Usually this is done if an expression has side effects or involves calling a function whose return value is to be discarded.

Escaped expression

= expression

The expression will be output to the stream with those characters replaced by their literal sequences, such as &lt;, &gt, and &amp;. Generally you normally do your work in this language expecting to generate HTML, but if you are processing a variable from outside, like H_from, which contains the E-mail addresss of a message -- which often has "<" and ">" in it, you need to take care about this.

IF statement

if ( Expression )
statement;
statement;
...
fi

and

if ( Expression )
statement;
statement;
...
else
statement;
statement;
...
fi

and

basic statement if expression ;

If the expression is true, the first set of statements are executed. If false, the ELSE set of statements, if present, are executed. This is the one statement that doesn't need a semicolon after it, though you can put one, since there is a null statement.

Note that we use a closing keyword syntax rather than a begin/end or brace syntax like many languages. Closing keywords tend to be less error prone.

You will use the IF statement to make conditional HTML or perhaps to have a template do entirely different things based on context or browser type or whatever else you need. You can also use it in combination with the RETURN statement to simplify templates that really need to do two or more entirely different things.

In the last form, a basic statement is anything other than another IF.

RETURN

Assignment

Variable = Expression;

Assigns the value of the expression to the variable, if the variable is not a system variable that can't be changed.

Not available in restricted templates.

Increment, Decrement:

Variable++;
Variable--;

These two statements increment or decrement the value of a numeric variable. Currently unimplemented.

Procedure Call

There is no procedure call per se. Most procedures simply don't return a value and as such can be called as functions. If you need to call a function that returns a value, use the discarded expression (:) statement.

Not available in restricted templates.

Misc Statements

PERM

perm Variable;

Makes the given variable permanent if it was temporary. Used only in complex applications for multi-queries and personal newspapers where you need to set values that will be used in another newsgroup or article.

Not available in restricted templates.

Expressions

Most of the ClariCGI programs consist of just expressions that get printed.

The most common expression is just a variable, but you can in fact have any mathematical formula, and some string formulae where needed.

Expressions are just terms (variables, constants and other expressions) combined together with operators. If you've done any programming this will all be pretty simple to you.

The two most useful operators you will use most of the time are the query operator (?) and the short-circuit-or (||) operator. You may also find yourself doing some string concatenation, pattern matching and the odd bit of arithmetic.

|| Operator

This highly useful operator has been stolen from C and Perl. It acts like the Perl version.

When you have a || b, the first expression "a" is evaluated. If it's "true" (non-zero or non-null) then the value is "a" and we don't even bother to calculate "b" at all. If a is false, however, we calculate "b" and the overall expression has its value. This is called a short-circuit OR in that it stops "b" from being calculated if it doesn't have to be, which is not only efficient but also stops side effects.

But the best use is to provide default values for expressions. This way you can refer to a variable, and if it's not present, provide another value to put in. So Subject || "hello" inserts the value of Subject, but if Subject is not defined it inserts the value "hello" for you. You can of course chain several together, so a || b || c || "hello" tries to insert first a, but if there is nothing there it tries b, and so on, until it puts in the default.

There is also an && operator which does the same, except that if the left operand is false, it stops immediately, does not evaluate the right operand, and provides a false result. If the right operand has side effects, they don't happen if the left operand was false.

Query operator (?)

The query operator is as found in C or Perl. It's an if statement in an expression.

boolean-expression ? trueexpr : falseexpr

The value of this expression is the trueexpr if the boolean expression is true, otherwise the falseexpr.

Unlike most languages, you can leave off the colon and false expression, and if the condition is false the result will be null/zero.

The un-needed expression is never executed so if you call a function with side effects, they will not happen.

HAS operator

String-Expression has Pattern-String-Expression

The second argument of this operator is a string that defines a regular expression. This operator returns true if the pattern matches the string.

The regular expressions are similar to those supported by Unix regcmp, but the actual package used is Henry Spencer's regcomp.

This operator also has the convenient form !has to reverse the sense.

String-Expression has [ number-expression ] Pattern-String-Expression

In this special form, the pattern string should have "tagged" regular subexpressions, which are surrounded in parentheses. The result of the operator is the substring that matched the tagged sub-expression. Sub-expressions start at 1, so if you ask for number 0, you get the substring that matched the entire pattern.

This is very useful for taking out components of strings or re-arranging them.

If the pattern doesn't match or a sub-expression doesn't match, you get a null string.

Other Operators

The following operators act pretty much like you have seen them in other interpreted languages, if you note the way strings and numbers can be interchanged:

+ - * / % < > <= >= != & ^ | 

Note that == and != can compare strings as well as numbers. The bitwise operators probably aren't that useful in typical templates, but they are there.

Operator precedence is similar to C. In order of precedence, from tightest to loosest:

() 
- ! 
* / % 
+ . - 
< > >= <= 
== != has !has 
&  
| 
^ 
&& 
|| 
? : 

dot (.)

The dot operator does a string concatenation. That even works on numbers, since they are interchangable. You get a string with the digits of the numbers concatenated together.

Not available in restricted templates.

Function Call

Function calls look like most languages:

Function-Name ( expr, ... )

Currently all functions are built in (see below). Some are meant to be procedures and always return zero (null).

A null argument list is valid. Up to 10 arguments may be provided. Parameters are passed by value.

Built in Functions & Procedures

The following special functions are built into the language. There is no difference between functions and procedures. Procedures just return zero so nothing will be output if they are used in the simple "expression" statement form.

body()

header( subject )

data( tag-name-string )

env( variable-name )

Not available in restricted templates.

fullname( fromline )

userid( fromline )

grouplist( groupline, [program, [delim]] )

Note that this routine is also the official way to make a read-the-group link for a single newsgroup as well as a list.

If a ClariCGI template program is provided as the optional 2nd argument, it will be used to output each entry. When it is called, the variable LItem will be set to the item (ie. newsgroup) and the variable LDelim will be set to ", " on the 2nd and later entries.

If the optional delimiter string is provided, it replaces the default delimiter set of commas and whitespace.

In this mode, you can actually use this function on more than just newsgroup lists, you can use it to make lists of anything.

msgidlist( references )

list( patternstring, level )

<LI>{if( Group_Min ) grouplist(List_Newsgroup); 
	%" - ({Group_Unread})"; else List_Newsgroup; fi}  
{if( Hier_Count )  
	%"<a href="{Script_Name}?;m+;g:{List_Newsgroup}\..*;v:"; 
	Hierarchy_Level+1; 
	%"\"><font color=darkred>[+{Hier_Count} groups]</font></a>"; 
	fi;} 

This is suitable for a list of newsgroups by name. The program is more complex if you have a DBM database of group descriptions.

If the level is non-zero, then the program detects hierarchies and provides them as a single link with the number of components in the level. Thus a level of 2 will list all 2nd level hiearchies (plus all 1st and 2nd level groups that match the pattern). The hiearchies will be linked to queries at the next highest level for that hiearchy. If a hiearchy is found, the variable Hier_Count will be non-zero on the listing for that group or template.

title( titlestring, ... )

query( querystring, ... )

Multiple queries can be provided, they are done in sequence. Recursion is aborted about 10 levels deep, and works but is not the recommended method for such operations.

This is the usual way to request that the menu of headlines be inserted into a newsgroup template. You invoke query( query_query ) to re-run the current query, but as noted in this case it only does the menu portion.

Template Hints

While the ability exists to define article templates on a per-newsgroup basis, this can sometimes cause problems because most articles appear in more than one newsgroup. This can cause the same article to have a different look based on what newsgroup it is posted in.

Queries can always define what template they wish to use if you have defined a directory for predefined templates and put some files there with template code. So if you don't find the customization capabilities of ClariCGI to be enough, you can always have any URL you write define the name of a template file with the "A:filename" query option and if you define that file in your template directory it will use it. This facility lets users invoke ClariCGI with any template in the directory, however, so be sure that you don't leave any security holes when writing such templates. Consider what might happen if somebody viewed an article with the newsgroup menu template you are writing. (Usually not much, the most the user will be able to do is get stupid looking output, not cause any damage.)

Users can also set variables that are not predefined or otherwise used by the system. So you should freely use variables instead of fixed strings wherever possible, using || to provide defaults if the variables are not set.

In order to keep the interprter and ClariCGI small, there are a few things about the language that are a bit confusing. Most notable is the ability to simple intersperse variables and code fragements into text at the top level. This does not exist at the lower level, except for the variable interpolation string output operator (%). This can be frustrating, and the string interpolation is more limited than the master level.

A guide to predefined variables is available and essential to writing templates.